Reinforcement Learning with Reusing Mechanism of Avoidance Actions and its Application to Learning Whole-Body Motions of Multi-Link Robot∗

In acquiring a motion only from its objective by learning, large cost such as damage from falling over and a large number of trials are required if the motion is a complex one, such as a jumping serve. Reusing the knowledge already learnt is an essential mechanism to learn such motions efficiently, like humans do. In this study, we propose to use a decomposition of action-value functions as a reusing mechanism for reinforcement learning. Avoidance actions that are assumed invariant across different tasks (e.g. avoiding to fall over) are learnt separately from primary actions assumed to be task specific. Then the action-value function for the avoidance actions is reused in learning new tasks. Furthermore, we extend the method for multi-link robots to learn whole body motions. This learning method has been applied for moving tasks both in discrete and continuous planes, and has been also applied for a tennis-serve and a jump tasks of a 4-link robot. The simulation results have demonstrated that in the moving tasks, reusing avoidance actions enables the agent to avoid the bad actions effectively. In learning whole-body motions, reusing avoidance actions obtained in learning a jump reduces to half the total falling damage in learning a serve compared to learning without reusing.